Implementing CER: What Will It Take?

BACKGROUND: Comparative effectiveness research (CER) is undeniably changing how drugs are developed, launched, priced, and reimbursed inthe United States. But most organizations are still evaluating what CERcan do for them and how and when they can utilize the data. A roundtable of stakeholders, including formulary decision makers, evaluated CER’s possible effects on managed care organizations (MCOs) and what it maytake to fully integrate CER into decision making. OBJECTIVES: To examine the role of CER in current formulary decision making, compare CER to modeling, discuss ways CER may be used in thefuture, and describe CER funding sources. SUMMARY: While decision makers from different types of organizations, such as pharmacy benefit management (PBM) companies and MCOs, may have varying definitions and expectations of CER, most thought leadersfrom a roundtable of stakeholders, including formulary decision makers,see value in CER’s ability to enhance their formulary decision making. Formulary decision makers may be able to use CER to better inform their coverage decisions in areas such as benefit design, contracting, conditional reimbursement, pay for performance, and other alternative pricing arrangements. Real-world CER will require improvement in the health information technology infrastructure to better capture value-related information. The federal government is viewed as a key driver and funding source behind CER, especially for infrastructure and methods development, while industry will adapt the clinical development and create increasing CER evidence. CER then needs to be applied to determining value (or cost efficacy). CONCLUSIONS: It is expected that CER will continue to grow as a valuable component of formulary decision making. Future integration of CERinto formulary decision making will require federal government and academic leadership, improvements in the health information technology infrastructure, ongoing funding, and improved and more consistent methodologies.

T o increase the understanding of what comparative effectiveness research (CER) means to formulary decision makers today and how the results of CER are expected to be used in the assessment of new technologies in reimbursement and benefit design, a moderated roundtable in formulary decision making for some executives from payers was convened in a session held coincident with the October 2010 Academy of Managed Care Pharmacy (AMCP) Educational Conference. The roundtable consisted of 9 representatives from a broad spectrum of health plan types as well as government bodies. This roundtable of thought leaders was invited to discuss the topic from their respective payer perspectives.
"Is the drug better?" and "Which drug produces the best outcome?" are key questions for the drug formulary decision maker that relate to efficacy and safety. Both questions require a direct comparison of the alternatives. The motivation of using CER to inform decisions may be driven by the need for utilization management: "How do we get the most out of the • Liraglutide has a direct drug cost of approximately $300-$450 per month of therapy compared with less than $10 per month for first-line drug therapy with generic metformin and $5 to $20 per month with the 3 sulfonylureas available in the United States as second-line therapeutic alternatives. • Currently, there is significant variability in the use of health economics outcomes research (HEOR) data for making coverage decisions. While the 9 roundtable participants indicated that HEOR information provided to Pharmacy & Therapeutics (P&T) committees is reviewed, when it comes to making a formulary inclusion decision, safety and efficacy from the randomized controlled trials (RCTs) and cost information from the manufacturer carry the most weight. • Comparative effectiveness research (CER) studies that compare a new drug to a variety of existing therapies, including nondrug therapies, would be of more interest to MCOs that have a broader view of patient care than to pharmacy benefit management (PBM) companies that are primarily concerned with drug-based therapies.

What is already known about this subject
• This roundtable of 9 MCO decision makers opined that the results of CER studies would be helpful in making formulary decisions at product launch and may be useful in the creation of coveragewith-evidence-development (CED) agreements in which coverage is provided under the condition that further evidence on the treatment's efficacy and safety is gathered over time. • In the absence of adequate CER data, modeling outcomes can better inform formulary decisions, as long as the model enables the decision maker to predict the impact of the drug in a plan's population in a transparent manner. However, CER data is still viewed as superior for the decision-making process, especially when it is available to be integrated into the model. • The 9 roundtable participants expected that formulary decisions will be more patient-centered in the future as compared to today where patients are not part of the decision-making chain. Currently, information on indirect cost, patient quality of life and patient opinion (or patient-group opinion) is not yet considered important for making formulary decisions. • The 9 roundtable participants opined that health-related quality of life (HRQoL) might be considered in formulary decisions, but the methods and data presented in HRQoL studies were generally considered insufficiently valid and reliable. • According to the roundtable participants, reassessments of medicines for inclusion, exclusion, or change in position in the drug formulary are currently rarely performed. Real-world CER can only effectively add value if reassessments become a standard part of the formulary decision process throughout the life cycle of pharmaceutical products. • The effective use of CER for understanding the performance of products in the real world will require significant improvements in health information technology and will require an evaluation of more effective ways to bridge health care sectors.
decision about the drug and its placement in the formulary.
In 2009, at the request of the U.S. Congress, the Institute of Medicine (IOM) published a prioritized list of research topics, derived from a broad stakeholder-input process that should be addressed by CER and funded by the American Recovery and Reinvestment Act (ARRA) of 2009. (See page S8 of this JMCP supplement for a discussion of the IOM report.) 2 The IOM definition of CER covers a very broad spectrum of research. However, there are many different players with many different perspectives (e.g., social, individual, and health-plan perspectives). It would be extremely difficult to solve all the interests with CER, especially because some of them may be contradictory. Even for a simple definition such as "the right medication for the right patient at the right time," there will be contradictory interpretations and conclusions. Some organizations or key opinion leaders only want to look at results of randomized controlled trials (RCTs)-"comparative efficacy"-while the IOM priorities apply to a much broader range of studies. For manufacturers, it is important to know and to get a clear commitment as to what kind of evidence is considered. The roundtable participants mostly agreed that there are advantages and disadvantages to all study types and that, depending on product maturity, different data sets might be relevant. Although there is a preference for prospective trial designs, retrospective data analysis was mentioned repeatedly as a useful way to assess the performance or issues in the membership population of a health plan.
Still, many of the 100 CER priorities defined by the IOM do not seem to generate information relevant for payer decision makers or, more specifically, formulary decision makers.

Current Use of CER in Formulary Decision Making
Consensus among the thought leaders at the roundtable suggests that available health economics and outcomes research (HEOR) data are always reviewed before making a formulary decision. HEOR assesses the clinical, economic and humanistic outcomes associated with health care interventions and can include observational, retrospective-data and decision models along with other related resources and tools. At the time of initial formulary decisions, there is usually a limited amount of real-world data available to the decision maker, and a majority of HEOR information will be supplied by the manufacturer. Once the drug is on the formulary, real-world data may be assessed within the membership population if the IT systems and analytical capabilities support such internal analysis. Several recent publications have drawn attention to the need for a more inclusive understanding of evidence when assessing the impact of technologies on health outcomes. [3][4][5] RCTs serve as the gold standard for evidence of safety and efficacy but may be limited in generalizability due to the population studied. Observational studies assessing treatment effectiveness in actual practice can complement the findings of RCTs and expand the generalizability of the findings to a broader population. resources in our budget?". If a new drug has to be assessed, the question will be "How does this drug directly compare with what is on formulary today?". Some organizations allow a broader comparison of available data. In systems in which members tend to remain for a long time, the view tends to be more comprehensive and less focused on the pharmacy silo. Specifically, the thought leader representing Medicaid felt that, in such an environment, CER helps to define value in the general health care context.
There are many proposed definitions of CER. Although all follow a common theme (comparison of how effective various medical treatments are at improving health outcomes), the perspective of the group or individual proposing the definition is important to consider. Two of these definitions, one from a pharmacy benefits management (PBM) company perspective and the other from a managed care organization (MCO) perspective, are discussed here.
PBMs, by the nature of their business, take a drug-focused perspective. Because they are responsible, for the most part, only for the drug budget for a health plan or employer, information about nondrug therapies is of little relevance to their decision making. CER is of interest to PBMs as long as it compares drugs or drug classes. This includes questions such as how a decision for 1 drug class will impact utilization of another drug class for the same or better outcome. The panelist with a PBM perspective thought that, within this framework, PBMs are interested in total drug-related outcomes.
MCOs have a somewhat different perspective. Because MCOs provide total health care coverage, their definition of CER tends to be broader, with a willingness to recognize that a higher drug cost may decrease overall costs to a health plan.
Decision makers will pick information from CER studies as it is relevant to them. In the short term, there will be no consensus-based way of using CER. Although cost-effectiveness has been excluded from the current definition of CER, cost will have to be brought into the equation for CER to be truly useful. CER must translate incremental cost to "value," or whether a drug is more cost-effective. If CER shows that a drug leads to better health-related outcomes, the decision maker must be enabled to value this improvement in comparison with the treatment cost for each therapeutic choice. Without this information, decision makers will default back to comparing net price per package of the therapies in question. A high price differential may intuitively override the incremental outcomes revealed in CER. For example, liraglutide has a direct drug cost of approximately $300-$450 per month of therapy compared with less than $10 per month for first-line antidiabetic drug therapy with generic metformin and $5 to $20 per month with the 3 sulfonylureas available in the United States as second-line therapeutic alternatives. 1 That cost information alone would lead payers to favor the established and cheaper alternatives. But if the CER incorporates cost-effectiveness research in addition to the health outcomes data, payers will be able to make a more informed real-world experience data are missing and will need to be collected over the next months and years when patients are using the medication.
Another challenge for integrating CER into decision making is the fact that post-launch data need to come from payers, and those data are currently not available in a single integrated manner. Furthermore, decision makers may not always agree with a CER study's design, inclusion/exclusion criteria, outcome measures (such as reduction in events vs. improvement in disease markers, primary vs. secondary outcomes, and so on), or other elements, thereby decreasing the applicability of that comparative data for decision making. A possible solution may be to include payers in a CER study's initial design phase, so that applicable data for decision making is more likely to be gathered and presented. Increasingly, as new drugs come to market with more comparative data for efficacy and safety, it is important to determine how the results translate into the membership population and their current standard therapies. In addition, cost must be factored in and, depending on the perspective and cost items included, the value (or incremental cost benefit) may not be as clear.

Current Use of CER for Coverage Decisions
Decision makers still make indirect comparisons on an ongoing basis. CER has the potential to make these comparisons more standardized and easier to execute. Also, in cases where CER data are not available at a product's launch, a conditional agreement based on a target for (comparative) effectiveness might help limit the risk to the decision maker and the patients.
While one-third of formulary decision makers in the national Internet survey stated that they are not using health economic or outcomes data for conditional reimbursement, another one-third claimed to use outcomes targets for conditional reimbursement agreements often, and the remaining one-third stated they rarely do so. 6 According to thought leaders from Medicaid, CER data are used for a variety of situations and for all new drugs in Medicaid. The primary question for the Medicaid organizations is not whether the drug will be reimbursed, but which is the best place in the formulary for the new drug. If the manufacturer participates in a federal rebate program, not covering the drug is not an option in the Medicaid environment, although controls such as prior authorization (PA) and step-therapy can be used to limit access. This is particularly true for newly approved drugs. Limited access allows the payer organization to gain some initial experience with the drug in a "protected" situation within the Medicaid patient population and to ask the manufacturer to bring additional data or analysis before the final place in the Medicaid program is determined after another review.
CER data may be useful in defining risk groups and limiting access to high-risk groups if safety problems arise. An example would be if a drug is more effective at controlling a specific disease (such as stroke), but it increases adverse events (such as serious bleeding). It may be appropriate to limit initial use to While all HEOR information provided to the P&T committee is reviewed irrespective of source (PBM, MCO, manufacturer, or literature), when it comes to making a formulary decision, safety and efficacy from the RCTs and cost information from the manufacturer carry the most weight. The results of a 2010 Internet survey, reported as a poster abstract, for 72 U.S. formulary decision makers (22 health maintence organizations [HMOs], 12 integrated health care systems, 26 PBMs, 8 preferred provider organizations, and 4 Veterans Affairs [VA] health systems) found that 73% of respondents said that they review HEOR data often or sometimes, while 14% indicated that they never review HEOR data in formulary decision making, and 13% responded they rarely review HEOR data. 6 In addition, only 32% of those reviewing HEOR data indicated that they applied quality standards to such data. This reflects significant variability among organizations in the use of such data for making formulary coverage decisions. Panelists noted that for HEOR data to increase influence on decision making, there would need to be a clear definition of HEOR data requirements (54%), more HEOR expertise on the P&T committee (47%), and a high need for in-house data analysis (42%). 6 The roundtable participants saw the key role of comparative HEOR data in deciding on the best place of a product in therapy in which "best place" could also mean exclusion of the product from the formulary. CER studies can be used to demonstrate the real-world performance of products; other applications of CER studies may be used to limit the target patient population, limit the financial risk and better understand patient behavior. The more information that can be assessed before making a decision, the easier it is to define the level of patient access to a product. Traditionally, clinical development of pharmaceutical products is planned with registration in mind, and data are often missing that would be useful for the decision maker. In other words, managed care would like to see data beyond those required to obtain drug approval. The acceptance of such data will grow as the data increase the ability of managed care decision makers to prioritize the use of therapies in the formularies, to reduce uncertainty, and to achieve maximum health outcomes in the reality of MCO membership mix, considering factors such as comorbidities, poly-medication, patient behavior, or adherence.
The greatest difficulty with CER is that it is usually not available when the drug comes to the market. In addition, there are limited incentives for providing CER data at launch. This is partly because a lack of U.S. Food and Drug Administration (FDA) guidance on what constitutes "competent and reliable scientific evidence" makes companies reluctant to make superiority claims even with supportive data. In some cases, it may also be because companies run the risk that CER data may clearly delineate the appropriate patient population that will benefit from a new medication, potentially reducing the market size for the manufacturer's new drug and thereby reducing revenues. Even if the clinical trials include CER, delivering data from RCTs include active comparators, the comparative well-defined endpoints and targets at the outset and that the time frame needs to be far shorter than in this first example for multiple sclerosis therapy. Often, however, it is difficult to define clear, measurable and nonambiguous outcomes targets that can be detected in a useful time frame, without putting additional burden on health care providers. An alternative to CED could be outcomes-based contracting. In an outcomesbased contract, the manufacturer is held responsible for the promised health outcome such as reduced adverse events, therapeutic response (e.g., A1c reduction), or less weight gain. The structure of such agreements could, for example, foresee a delayed payment after reaching the target outcome or a credit note by the manufacturer if the target outcome is not achieved by the individual patient or in a patient population. However, outcomes-based contracting requires an electronic infrastructure, in which conditional outcomes-related payments can be administered through automated functions and feedback routines. Currently, the complexity of outcomes and data tracking make the implementation of outcomes-based contracting impossible for most plans.
From the manufacturer's perspective, a conditional reimbursement agreement may help to overcome decision makers' fears regarding the uncertainty surrounding the new product, but it may also set a precedent for rebates. Once the manufacturer has agreed to a rebate on a conditional-access basis, it may be difficult to withdraw this rebate later.

Usefulness of Models to Estimate CER and Cost
At the time of product launch, there is a strong desire for more economic information and more information about the impact of a new treatment versus existing alternatives. However, this information cannot be delivered without the use of the product in the real-world health care setting. Modeling economic and the population less at risk for the adverse event. On the other hand, it would be very difficult to take drugs off the formulary if they do not perform as expected or intended. Nonetheless, there are published examples of drugs that have been taken off the formulary-usually because of new or additional safety concerns, or because a generic alternative became available that facilitated placing the brand drug in a nonpreferred position. 7 The more alternatives that are available in any indication, the easier it is to discontinue reimbursement for a specific drug, especially if there is no CER proving that the more expensive drugs work better or are cheaper when considering total health outcomes and total cost of health care.
Towse and Garrison (2010) attribute a number of deviceand drug-reimbursement agreements with the Centers for Medicare & Medicaid Services (CMS) to the coverage-withevidence-development (CED) concept. 8 Under CED agreements, reimbursement coverage is provided by CMS under the condition that the patient is enrolled in a registry in which further evidence on the treatment's efficacy and safety is gathered over time. 9 An early international example of a CED payment scheme was the outcomes-based agreement between the United Kingdom and manufacturers of interferon that treatment for multiple sclerosis would be covered with the understanding that data would be collected and evaluated later to make a final coverage decision. Initial evaluations of the 10-year data were published in 2009 and showed poor results for drug therapy. This dramatic finding, however, did not trigger any price reduction. 10 Instead, the paper reports, "The scientific advisory group considered that it was premature at this stage to reach any decision about re-pricing the drugs without further follow-up and analyses." 10 Such experiences have taught decision makers and manufacturers that conditional reimbursement agreements need Implementing CER: What Will It Take?

Key Factors to Consider When Building a Model for the Use of Formulary Decision Makers a factors Concerning Model Development
• Author/creator of the model needs to be identified • Methods of quality control and validation • Transparency of the model • Transparency of the factors that were included and excluded • Any budget impact model must be available to the decision makers to populate with data from their own member populations • There is no gold standard for how to make assumptions, but assumptions are an essential component of models. This uncertainty makes adapting the model conclusions to the needs of multiple stakeholders even more complicated (for example, for PBMs helping their various customers to make decisions) factors Concerning the Population in the Model • A change of the model population for demographic-or disease-related factors must be possible to make the results relevant to the local health plan • The more a plan covers a nonstandard membership, the more important it is that a model is all-inclusive so as to be a useful indicator of what might happen • High transparency and flexibility of the model, to allow combination with the plan's electronic data systems, will increase acceptability among decision makers in such plans • Models enabled to analyze subpopulations can be used by the plan to estimate the impact on specific patient segments (for example, single mothers, elderly people, disabled people) a These factors were defined by the 9  clinical outcomes based on data from the experimental study designs is an option used to simulate the impact of the new product, thus enabling decision makers to estimate what overall outcomes may be expected.
Still, debate among thought leaders continues over whether modeling clinical or economic outcomes can better inform formulary decisions. About half of the thought leaders in the roundtable discussion would consider data resulting from validated models, while the others remained skeptical. The CORE Diabetes Model, which was used to project and compare lifetime clinical and economic endpoints for liraglutide or rosiglitazone therapy added to glimepiride in the treatment of type 2 diabetes, was one model that the panelists mentioned as believable. 11 The properties of a model may determine whether it is perceived as useful and relevant. The roundtable participants suggested factors considered important when evaluating a model (see Table 1). The essential point is that a model must enable the decision maker to predict the impact of the technology in a plan's population in a transparent manner. However, the thought leaders felt that a model will likely not eliminate the need for CER, unless there would be a very high trust that the model reflects reality.

Future Role of CER in Formulary Decisions
Seven of the 9 thought leaders of the roundtable discussion thought that in 5 to 10 years, CER will always be used when making decisions about adding new drugs to a formulary. Among U.S. formulary decision makers from a broad range of payer organizations in the national Internet survey, the majority (81.9%) expected an increasing influence of HEOR data in the future, while only 2.8% of the 72 survey participants expected no increase in use (and 15.2% expected only limited use). 6 However, it seems that the organizations are not yet fully prepared to use HEOR or CER information. There are concerns about the quality of the evidence because unlike an RCT with accepted methodologies, HEOR and CER methodologies are still evolving. Participants in the national survey and the roundtable discussion found it very important to obtain clear definitions of the role and use of CER and HEOR data in their organizations. In addition, both groups of decision makers agreed that a higher level of in-house competence would be needed in their health plans, MCOs, PBMs and hospital management to be able to analyze their own data and to interpret the results.

Which Outcomes Data Will Be Important?
When asked what type of information is important for making formulary decisions with a focus on diabetes, the 9 thought leaders at the roundtable ranked the following criteria highest: the risk for adverse events, medication adherence and persistence, and health care utilization. (Again, adherence and persistence cannot be shown until the medication has Implementing CER: What Will It Take? been available for an extended amount of time.) Medication adherence and cost information were deemed more important by the thought leaders than HRQoL data, and with regard to economic information, cost-effectiveness was rated above budget impact and direct medical cost data. None of the panelists believed that societal cost (for example, caregiver time or patient productivity) would play any major role in formulary decision making.
While medication adherence may not be the most important factor for decision makers when it comes to making formulary decisions, it is nevertheless an important factor in health plans or PBMs to improve utilization of all formulary medications once coverage criteria have been defined. For example, 1 major PBM has its own therapeutic research center, the major goal of which is to increase medication adherence. The thought leaders also suggested that patient copayment levels could help to achieve goals (e.g., via lower payment with consistent adherence or weight loss). While decreasing their premiums, patients can be more responsible for their own health. Adherence is something that PBMs could measure with their data, and this activity could make a difference in patient outcomes. The value of looking at adherence data is somewhat tempered by the fact that adherence in most drug classes is far from ideal. Realworld comparative compliance or adherence data may be used to determine continuation of coverage or preferred formulary status upon subsequent class reviews.
Still, safety and efficacy data from active-comparative RCTs (RCT-CER: comparative efficacy research) and observational CER studies (actual practice comparative effectiveness) are priorities and will continue to drive decisions. In addition, real-world CER and HEOR offer a broader body of evidence. However, since evaluating HEOR data takes additional time and expertise than evaluating RCTs, it will not be used for all products or therapeutic classes. A higher degree of standardization for the generation of all outcomes data, as well as for the use of these data, will support a broader adoption of such data in standard assessment procedures.
An additional dimension in determining the value of product could be patient-reported outcomes including, for example, HRQoL, patient preferences or patient satisfaction. Despite public discussions on increasing consumerism in health care, patient satisfaction was not deemed important for better decision making by any of the thought leaders. Patient satisfaction is often measured on a general level through surveys by payer organizations but is not relevant to understanding the effectiveness of a specific treatment. Patient satisfaction may be relevant to consumers selecting a health plan but is not currently influencing formulary decision makers on drug selection. In the thought leaders' opinion, HRQoL would be a more relevant measure, but because methods or standards are not sufficiently defined and robust data are difficult to compare, quality of life is not usually a key consideration. Improvement might mean including more than 1 endpoint or even a standardized list of outcomes (e.g., cost, short-and long-term clinical outcome, out-of-pocket cost, adherence or persistence). The authors felt that these programs need to control medication use and lower overall medical costs without hindering patient outcomes over the long term, as opposed to the short-term benefits displayed by many.
Several studies have attempted to analyze the impact of benefit design on patient behavior. For example, an analysis of retrospective claims in 2001-2002 performed by Brixner et al. (2007) found that a change of benefit design involving a pharmacy benefit design change (BDC) was associated with medication adherence for 4 of 5 drug classes studied. 14 BDCdefined as a copayment increase in the second or third tier of $5 or greater or as a change from a flat copayment to a percentage coinsurance (intended to raise the incentive to use generic medications and formulary brands by increasing the copayment differential between generics and tier 1, between formulary brands and tier 2, and between nonformulary brands and tier 3 by increasing the tier 2 copayment or tier 3 copayment or both)-compared to a group with no BDC was associated with a higher proportion of patients who discontinued drug therapy (67% vs. 54% for allergic rhinitis), (66% vs. 50% for asthma), (61% versus 36% for osteoarthritis) and hypertension (39% vs. 18% for hypertension; P < 0.05 for all). BDC was not associated with discontinuation of diabetes drug therapy, and BDC was not associated with reduced adherence measured by the medication possession ratio (MPR). The BDC groups' year-to-year pharmacy costs per patient were lower in 2 of the 5 drug classes, but there was no effect of BDC on total overall health care costs in any of the 5 drug classes. 14 The roundtable thought leaders commented that, ideally, a reassessment of formulary decisions should be performed, but reassessment is rarely done. When a new drug is launched, an initial decision on preliminary inclusion or exclusion from the formulary is usually made relatively quickly using the best information available at the time. Once sufficient realworld data become available, the thought leaders believe a reevaluation should take place to confirm or revise the decision. According to the thought leaders, this approach would result in a transformation of the contracting process, acknowledging the lack of sufficient relevant data at launch and encouraging manufacturers to develop additional data during the postmarketing period. The decision will be delayed by 6 months or longer if needed, and meanwhile, limited market access will be granted.
A similar approach was suggested recently as a future model for CMS. 15 Medicare would cover and reimburse an innovative new technology or new drug therapy, as it does today, for a maximum of 3 years. During that time, manufacturers and clinicians would have to carry out research to compare the performance of the new treatment with the prior standard of care in the real-world setting. If evidence showed that the treatment did not offer clinical advantages, CMS would cut

The Role of Patient-Centered Outcomes Research Institute (PCORI) Data
PCORI's patient-oriented objectives will give more weight to the patient perspective of CER. However, according to the payers represented in this roundtable, it remains unclear whether research from PCORI will be useful and timely for better formulary decision making, since the PCORI's research priorities do not necessarily match payer priorities. In addition, as stated by decision makers, certain patient priorities (such as the humanistic outcomes like quality of life, productivity, and caregiver burden) and societal costs are not their concern and will not influence their decision making.
Cost remains a crucial factor for decision makers as long as they have to balance the use of limited resources. However, a current provision in the Patient Protection and Affordable Care Act of 2010 (PPACA) reflects the political nature of this subject, and thus, PCORI may not have the responsibility or even the authority to measure value as determined by cost per outcome. 12 In the equation used to calculate value to an organization, PCORI may help to better define the desired clinical and patient benefits. However, PCORI is not expected to define the cost of these benefits or whether the benefits are affordable.

CER Data in Formulary Design, Benefit Design, and Re-Evaluations
Today, decision makers do not consider further assessment a priority after the primary decisions on inclusion, exclusion, and position of a drug in the formulary have been made. With a few exceptions, the impact of drug-formulary or benefit design decisions on health outcomes is generally not measured. 6,13 As 1 thought leader noted, "We rarely go back and see what the impact of our decision was." This emphasizes the issue described earlier about the lack of real-world CER data at a product's launch. If there is a desire for more real-world CER data, then decision makers must use the data and make it a priority to follow up on class reviews.
Benefit design may also impact the development and utility of real-world data and retrospective analysis and re-evaluation. This observation is confirmed by a systematic review of publications that analyzed the impact of benefit designs on health and economic outcomes. 13 Seventy-seven studies were included in the review by McAdam-Marx et al. (2008) and were analyzed for the use of 11 types of outcome endpoints. Sixty-eight percent of studies incorporated an economic endpoint; of these, 68% reported only economic data and did not address clinical or humanistic outcomes. Overall, clinical or humanistic endpoints were evaluated by 43% of studies; of these, an additional economic endpoint was reported by 52% of studies. 13 The authors concluded that "the efforts of these researchers to assess the overall quality of drug-management programs have fallen short. To ensure that drug-management tools have a desired effect on outcomes and medical costs, measures used to evaluate drug-management programs must be improved." 13 HEOR data are already used in contracting, and up to twothirds of respondents believe their use is expected to increase in the future. 6 Outcomes-based contracting will most likely be used to cover gray areas where there is insufficient information available at the time of decision. For example, to make it more cost-effective, a new drug may have to offer larger rebates until CER is available. Contracting affects the value equation for a drug. Additional use of real-world CER in contracting will require improvement in the health information technology (HIT) infrastructure-specifically, improved electronic medical records and data integration across sectors. This alone is an enormous challenge. It is not enough for each physician's office to maintain a patient's medical records, but those records must also incorporate patient data from other sites of care, wherever that patient is seen. Every physician's office and hospital must share patient data and communicate with each other. In addition, financial incentives across sectors need to be aligned. A PBM and a MCO have very different financial incentives, for example.
Improved HIT may help to better capture value-related information. When this information is available, outcomesbased contracting will be more operationally manageable. In addition, the same outcomes targets and criteria could become the basis for new disease-management programs.
Outcomes-based contracting or outcomes-based disease management will work better when the long-term-oriented payer systems are more standardized. On the contrary, in organizations with a multitude of different customers with different needs and patient populations, as in the case of PBM companies, it will be more difficult to determine the targets for each customer and to then differentiate among various contract types.
There is an additional dimension of outcomes-based contracting: To capture the full potential value of a therapy, decisions will increasingly have to bridge health care sectors (e.g., pharmaceuticals vs. physical therapy vs. surgery). The current budget structures and incentives limit the usefulness of cross-sector comparison. In addition, many plans and PBMs currently don't have the necessary integration of medical and pharmacy data to make this type of integrated contracting feasible and would need better HIT.
For practical reasons, not all data can be generated locally for each plan or system. Decision makers will, however, be interested in seeing how applicable the data are to their own populations and how to improve outcomes in their own populations. In addition to using drugs in the right patients (highchance responders), alternatives are needed to improve the chances for patients at high risk for nonresponse. Thus, targeting and outcomes-improvement programs could be additional aspects of CER.
CER can support contracting on 2 levels. Using CER with the supply side would mean that the payment depends on concrete and measurable outcomes. The contracts will mostly rely on health outcomes (clinical parameters). According to the 9 formulary decision maker thought leaders in our roundtable, the price to the level it pays for the equivalent, conventional treatment. But, if the evidence showed that the new intervention was superior to the traditional method, Medicare would continue to pay the higher rate. 15 Where will CER lead formulary decision makers? The majority of thought leaders (7 of 9) in the roundtable discussion agreed that benefit design will increasingly consider CER data over the next 5 to 10 years. CER will probably not tip the scales for or against coverage, but it is expected to help in finding the best place in therapy and on the formulary (e.g., copayment tier, PA, or step therapy).
With all the investments in CER in the United States, it can be expected that in a few years more CER data will have been published and be available. Comparative research is also available from other regions worldwide and may add to the evidence base. While this information will be useful, it will not necessarily change the process of how decisions are made. Even if health care reform stagnates in the United States, some changes, such as the advances already made in CER that provides clearer information for decision makers, cannot be ignored. The more that CER data become available, the higher the expectations of decision makers will be. Thus, the main change will be in the intensity of how the data are used.
However, the use of CER data will not only be accelerated by payer demand or government policies. Each additional head-to-head comparative study submitted with a new drug in a therapeutic area or class will raise the expectations for future development of subsequent drugs. Because of pressure from early CER adopter manufacturers, other manufacturers will follow, despite the fact there are limited incentives for providing CER data at launch from the registration and marketing perspective.
Eventually, with more experience and standardization, and as both RCT-CER and real-world CER become more abundant, CER will become more important and a fully integrated aspect of formulary management. While, initially, there might be more pressure on new drugs with perceived high-cost consequences (including drug costs, costs of adverse events, costs of monitoring and utilization) for the health plan, CER should be evaluated for all drugs on the formulary.
In addition, CER may offer the opportunity for earlier market access under the condition of continuing data collection in the real world: coverage-with-evidence development (CED). Under such arrangements, the drug is reimbursed, but the manufacturer has to build a registry to collect data documenting the course of the disease in patients treated with the new product. As CER evolves and becomes a more important component of decision making, the impact of benefit design on outcomes will become more important in CER design.

CER's Effects on Health Information Technology, Contracting, and Bridging Health Care Sectors
According to the (unpublished) national Internet survey of formulary decision makers, one-third of the respondents felt Implementing CER: What Will It Take?
A very small percentage of ARRA funding for CER has been earmarked for understanding the genetic or genomic basis of diseases or response to treatment ($4.8 million or 2.5%). 16 At this point in time, the research projects funded by the National Institutes of Health (NIH) are not necessarily matching up against the priorities set by the IOM. The highest 4 NIH expenditure areas for evidence development and synthesis are psychiatric disorders; oncology and hematology; cardiovascular and peripheral vascular disease; and alcoholism, drug dependency, and overdose. The IOM's top 4 research priority areas are functional limitations and disabilities, cardiovascular and peripheral vascular disease, psychiatric disorders, and neurological disorders. 16 The debate continues as to how well either of these priority lists align with those of health plans and formulary decision makers.
According to the 9 roundtable participants, government is playing a leading role in defining the cornerstones for CER. Many other players are still puzzling over how CER applies to them and are waiting for some government or academic leadership. An immediate and extreme change-legislation requiring companies to conduct clinical trials against a current standard of care, for example-is unlikely in the short term because of the combined mass and inertia of both the industry and government. Marketplace demand will probably drive CER's growth, at least in the beginning. Pharmaceutical companies may be a secondary driver of CER. As a result of CER becoming a general expectation, new products in development will have to fill the gap in CER by integrating comparative trials into their study programs. Comparative clinical studies have been supplied with an increasing number of drug applications in recent years. 20 With the increasing request for comparative data, some drug companies proactively take on a leadership role in designing active comparative trials (e.g., with liraglutide). [21][22][23][24][25][26] Based on such studies, formulary decision makers will have more comparative efficacy data on hand for newly developed drugs to support their decisions. At the same time, the bar of expectations will be raised and active comparator trials will be expected if the study designs and data produced are judged to be valid.
In addition to funds coming from the government, some funding for CER will come from pharmaceutical manufacturers who will voluntarily invest in active comparator trials as a competitive business decision. Increasing experience with manufacturer-driven CER will begin to shape how CER is integrated into drug development. Moreover, there are voices requesting CER to be mandatory for FDA approval. [27][28][29] Health plans do not have the resources to fund CER. They will require it, but the thought leaders at the roundtable discussion felt that government rather than any other stakeholder will have to be the power behind the impetus for change, potentially through CER mandates. Making the right investment of government funds, defining the appropriate health targets, and deciding on the acceptable value limits in terms of cost and humanistic outcomes such as HRQoL can only shift the weight of a decision in the United States. Quality of life is not expected to become a key decision criterion to the payer, and payers are not expected to pay an extra margin based on quality-oflife outcomes. Similarly, patient time or productivity is not expected to be highly valued.
Using CER for contracting with the user side would mean that the amount of copayment contributed by the patient will depend on achieving goals (such as consistent adherence or losing weight). The purpose of these types of contracts is to motivate the patient to be responsible for his or her own health. More re-evaluation will need to be undertaken and made a priority. Value-based benefit designs have been in use for some time, but more data are needed to evaluate the clinical and financial benefit to the payer organization.
In disease areas like type 2 diabetes, hypertension, and obesity, increased patient participation in the cost may appear attractive to the payer organization. The perceived benefit of cost-sharing programs directed toward the patient could be that the patients become more motivated to adhere to the therapy, and thereby, better outcomes are achieved. The expected benefit of contracting with drug manufacturers based on clinical outcomes would be that the financial risk for the payer is limited to patients who respond to therapy.

Who Will Initiate and Fund CER?
Several sources of funding are available to support the establishment of CER. Additionally, future funding has been promised by the president of the United States and the PPACA. [16][17][18][19] Sources of funding include: • In 2010, $1.1 billion in stimulus funding for CER was made available through the ARRA. A considerable share of the original funds was used for infrastructure and methods development. 16 • The Agency for Healthcare and Research Quality (AHRQ) fiscal year 2011 budget request included $286 million to broaden patient-centered health research. 17 • The PPACA provides sustained federal funding for CER through 2019. 19 Forty-six percent of the $1.1 billion funding has been allocated to synthesizing existing evidence through systematic reviews or developing new evidence. A further 41% will be allocated for CER to develop the data, methods and workforce that can, in turn, increase the capacity for future research. The majority of spending in both of these categories will be dedicated to using the knowledge acquired from the delivery of care or observational research. 16 Moderate percentages of the funding were dedicated to: • Comparing models for the integration and coordination of care ($38.6 million or 15.1%) • Studying the delivery of care in nontraditional settings or by nonphysician providers ($34.3 million or 13.5%) • Evaluating economic incentives for patients and providers to choose effective treatments ($8.0 million or 3.2%) 16

DISCLOSuRES
This JMCP supplement was sponsored by Novo Nordisk Inc. and prepared by the Millcreek Outcomes Group (Salt Lake City, Utah). Joseph Biskupiak reports receiving payment as a manager of Millcreek Outcomes Group for his contributions to this JMCP supplement. Jeffrey Dunn reports receiving compensation from Millcreek Outcomes Group for this project and supplement and for participation on an advisory board for Novo Nordisk Inc. He also reports compensation from Biogen Idec for a Payer Steering Committee and consulting fees from Genentech, UCB, and Teva. Anke-Peggy Holtorf reports receiving payment as a contractor of Millcreek Outcomes Group to participate in a thought leader discussion on Comparative Effectiveness Research in Decision Making, to survey the use of health economics and outcomes research by U.S. decision makers, and to contribute to this article; she is a member of the International Society of Pharmacoeconomics and Outcomes Research (ISPOR) HTA special interest group and a member of the Health Technology Assessment International (HTAi) Association.
appropriate outcomes would be the government's responsibility. From the perspective of the pharmacy formulary decision makers, the real question is whether the tremendous investments in CER through stimulus funds and expected tax contributions, along with the accompanying changes in the investment priorities of health care-related organizations, will really lead to studies that deliver results with a higher impact on health care efficiency. A review of recent CER trials suggested that poor quality research continues to emerge and that there are methodology issues that must be addressed to improve the degree of usability of CER. 30 This includes many aspects of the studies such as the selection of less-than-optimal comparators, inadequate dosages of comparators, the use of surrogate study endpoints that may or may not have been validated as clear indicators of specific clinical outcomes (e.g., using biomarkers such as low-density lipoprotein [LDL] cholesterol levels for coronary artery disease), and accelerated time frames. 20 The PCORI is expected to assist in defining criteria for CER trial design in the future.

■■ Conclusions
As noted by the roundtable participants, both RCT-CER and real-world CER studies are highly desirable for formulary decision making, and both are used when available. However, such research is not yet routinely available, and it does not yet apply to all formulary decision-making organizations and is not yet used routinely. It is expected that the role of real-world CER will increase as more results are published and accessible, and as the guidance and training on how to use real-world CER research and results emerge. The government is viewed as a driver behind CER, while the pharmaceutical industry will shape it by adapting the clinical development process and creating increased evidence from both pre-and post-approval CER. To make it sustainable, however, decision makers will have to require CER evidence. For formulary decision makers, CER is useful only if it has application. Ideally, health plans will develop processes to effectively conduct, analyze, and use CER data to understand the results in their own populations, enabling formulary managers to create effective benefit designs or make better formulary decisions. CER must then be applied to determining value (or cost-effectiveness).
The evidence generated in RCTs should be validated in real life to show that the expected advantages are also seen in realworld populations of various compositions. Decision makers need to improve retrospective analysis of data within their own health plans to ensure validity of their decisions.
The roundtable panel recommended that manufacturers maintain close communication with decision makers when planning and gathering further evidence in real-life studies and that such communication remain proactive and transparent.